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ABSTRACT 


Recent years have seen many attempts to program learning 
based on many principles ranging from intuition to mathematics. 
The area of criterion establishment for advancement or con- 
versely additional training at the same or lower levels remains 
primarily in the realm of intuition. The intuition used is 
only as good as the experience and background of the program 
designer however. In the case of a very experienced program 
designer the criteria may be very efficient. In the case of 
less experienced program designers the criteria are usually 
arbitrary. This work describes a method of analyzing learning 
programs and determining mathematically sound criteria. The 
mathematical foundation for this analysis is the Markovian 


learning model as opposed to the linear learning model. 





TABEE OF CONTENTS 


I. INTRODUCTION ------ see 2 === _ 6 
A. GENERAL ------------------------------------ 6 

3 DRO 7 
Cs 7 

II. MODEL DEVELOPMENT ------------------------------ 9 
A. BACKGROUND -------------- Se eee __ 9 

Pe oe ee ee abil 

C. MODEL ----------------------9----- nh 

III. APPLICATION OF THE MODEL rerrrrr ren << e222“ ------ Ie, 
2. BEER S00 1c SSS Se ee ee ee ee 19 

B. EXAMINATION OF THE DATA) --92---99----27------ 20 

C. CRITERIA DETERMINATION ------2220------------ 24 

1. Advancement Criteria ------------------- 24 

2. Branching Criteria --------------------- 26 

IV. CONCLUSION) --- 2-99 230-9 en ee 32 
APPENDIX A: Individual Data Matrix (T and E) -------- 34 
me eeENDEX Bs Individual Data Matrix (Gaandac woe =2e4-— 40 
LIST OF REFERENCES) 2-23-99 e ere e en nnn 54 
INITIAL DISTRIBUTION LIST ---------------------------- ay. 
FORM DD 1473 --<-------------+-= --- = - - 59 





eg 


Pod. 


IVE 


LIST OF TABLES 


Mean and Standard Deviation of g and c by 
Subject ------------------------------------- 22 


Q> 


Mean and Standard Deviation of g and ¢ by Step - 23 

Estimated Probability of Not Being in Solution 
[P(S)] Once the Criterion of X Consecutive 
Correct Response Has Been Achieved ---------- ISS 


Estimated Probability that the Number of Errors 
Will Exceed a Given Criterion k [P(E>k)] ---- 29 


Estimated Probability that the Number of 
Trials Will Exceed a Given Criterion t 
[P(T>t)] ------------------------------------ 30 





DIST Or er Gures 


Frequency Distribution of the Total Number 
of Trials Prior to and Including the Last 


Error (T) -------------------------------- -- -- == ++ 


Frequency Distribution of the Total Number of 
Errors Prior to and Including the Last Error (E)- 


Frequency Distribution of the Estimated 


Probability ‘of Going into Solution (c)------------- 


Frequency Distribution of the Estimated Pre- 
Selui~don Probability of a Correct Response 


or Guess (9g) core -- oe ne ee eer 


Frequency Distribution of the Mean Values (by 
Subject) Of Lhe Estimated Probability of Going 


into Solution (c) -------------------------------- 


Frequency Distribution of the Mean Values (by 
fe) Orethe bstimated Probabality of Going 


into Solution (c) -------------------------------- 


neequency Dagetrabution of theeMean Valwes (by 
Subject) of the Estimated Presolution Probability 


of a Correct Response (g) ------------------------ 


Frequency Distribution of the Mean Values (by Step) 
of the Estimated Presolution Probability of a 


Correct Response (g) ----------------------------- 





I. INTRODUCTION 


A. GENERAL 

This work was stimulated through the author's association 
with the Behavioral Sciences Institute of Monterey, California. 
Meese Ys of the staff of the Institute have developed a pro- 
cedure for preparing language training programs [Gray and 
Ryan 1973]. This procedure Caner Tramnang slhvough 
Programmed Conditioning" is based on a mathematical principle 
or structure. The total language program consists of ninety 
Sub-programs covering different language deficiencies which 
are the result of various causes such as deafness, mental 
retardation, non-English speaking, etc. Each sub-program 
consists of a series of steps that sequentially train a 
Student until he reaches a desired objective. The total pro- 
gram is designed to test the student in a continuous manner 
to determine what sub-programs or parts (steps) he needs, 
when he has completed any step or complete sub-program, and 
when he is having excessive problems that require stepping 
back (branching) or other forms of special help. 

The procedure, though developed for language training, 
probably has wide application. These applications will appear 
aS more people, in varied fields, become familiar with the 
procedure and investigate its appropriateness to their svecific 
areas of interest. Along this vein it is also hoped that the 


method developed herein, of analyzing the program and 





determining criteria to be used in the administration of the 


program, will have wider app lbacation. 


B. PROGRAMMING 

The objective of an intellectually oriented training 
program is to move a subject from a specific unlearned state 
(U) to a learned state (L). The process used to achieve this 
goal is Stimulus-Response-Consequence, normally referred to 
as operant conditioning. In most situations the desired 
learned state (L) can be achieved most effectively by reducing 
the unlearned state (U) to a number of small steps (states) 
which are learned in a sequence and build up to the ultimate 
goal. This latter process is called programming. Throughout 
this work the term presolution will be used interchangeably 
with the unlearned state and the term solution with the 


learned state. 


C. CRITERIA 

The subject of criteria arises when the programmer desires 
to measure the success of a given program in moving a specific 
subject from the unlearned to the learned state. Basically the 
guestion is: "What measure should be used for determining 
when a subject has achieved the learned state (L)?" A second 
aspect of criteria arises when it appears that a subject is 
having an excesSive amount of trouble with a particular step. 
In such a case it may be desirable to alter the program for 
the subject, possibly reducing the step to several less dif- 
ficult substeps. The first criterion is referred to as 


advancement criterion and the second is referred to as 





branching criterion. A difficulty in establishing these 
criteria results from the generally accepted idea that there 
is no exact (deterministic) method of establishing criteria. 
Alternately learning 1S viewed as being probabilistic (sto- 
chastic) in nature and therefore the criteria must be 
established within the realm of probability theory. To 
further complicate the situation there is not a universally 
accepted learning theory and therefore the criteria, if 
determined statistically, will be a function of the learning 
theory the programmer subscribes to. 

Some possible measures for criteria are: total number of 
correct responses, total number of incorrect responses, total 
number of trials, plus the associated ratios and percentages. 
A trial is one Stimulus-Response-Consequence sequence. Of 
these it seems intuitively epee to select total number 
or percentage of correct responses as an indication of when 
to advance a subject. More specifically it 1S common practice 
to use a string of consecutive correct responses for advance- 
ment. Determining the number in this string and the confidence 
in that number is required in establishing the first criterion. 
Determining a maximum acceptable number of trials and/or 
errors per“step, and the associated confidences, 1s the pro- 


cedure used to establish the second criterion. 





If, MODEL DEVELOPMENT 


A. BACKGROUND 

In a brief historical review of mathematical learning 
theory, Atkinson, Bower, and Crothers (1965, pp. 19-24), 
iGiicate that the treatment of learning uSing probabilistic 
models began with Thurstone in 1919. From 1919 through 
1950 there were numerous probabilistic models propcesed and 
tested, all related to various specific learning situations. 
Since 1950 there has been a great deal of work done in the 
area of stochastic learning, and two theories have evolved 
as the primary contenders for acceptance: the linear model 
and the Markov model. Basically the linear model states that 
a subject has a probability of Success on any trial n which 


is given by the following equation: 


P= 1- (1-P,) (1-6)"™* (1) 


where Py is his initial probability of success and 6 is his 
learning rate. The Markov model takes a quite different 
approach. Basically the theory says that if a subject is in 
an unlearned state (U) then the probability of a correct 
response is g (guess), and if the subject is in the learned 
State (L), then the probability of a correct response is l. 
The probability of going from the unlearned state to a learned 
state on any presolution trial is usually denoted c. The 
probability of a correct response on any trial n is usw: tly 


given by: 





Poo = 1 - (1-g) (1-e)?™ (2) 


Comparison of Eqs. (1) and (2) indicates that their forms 
Bee exactly the same. The difference in these two equations 
lies in their theoretical background and the meaning of the 
parameters. Equation (1) states that a subject starts with 
a probability Py of making a correct response on the first 
trial. The probability of success on the second trial is 
greater, due to the incremental learning achieved on the first 
trial. This linear process continues indefinitely and the 
Subject's probability of Success asymptotically approaches l. 
Equation (2) states that on each presolution trial a subject 
has a probability c of going into solution. Once in solution 
the subject will remain in solution and always respond cor- 
rectly (P=1). While in presolution the subject has a proba- 
bility. g (guess) of responding correctly and this probability 
does not change. The form of these two equations and other 
interesting comparisons are made in "Learning: Gradual or 
All-or-None," [Restle and Greeno 1970, Chap. 2]. Based on 
their analysis Restle and Greeno stated, ".. . the all-or- 
none theory is the most interesting, and we think it is the 
one most deserving of future work," (p. 79). 

Pilot research involving a computer simulation of the 
linear model yielded a set of criteria for use in the language 
training programs of the Institute. Informal observation of 
the data on which the linear model was based suggested that 


the linear model might not be appropriate. Further 


1B 





study CrsG@eranrtrem Students trained on these and other 
programs showed a definite tendency toward the Markov 
principles of stationarity and independence [Atkinson, and 
others, 1965, pp. 39-49; Coombs, Dawes, and Tversky 1970, 
pp. 296-297]. Based on these results the linear model was 
set aside and this work started based on the Markovian (All- 


or-None) principle. 


B. ASSUMPTIONS 
“The following assumptions are necessary to the development: 

1. The learning process involved in the programs devel- 
oped by the Institute are Markovian in nature. 

2. The subject can be correct on the first trial of 
payee by either: (1) betng in solution prior to the trial, 
(2) going ee solution because of the information presented 
in the first stimulus, or (3) guessing correctly in presolution. 
This assumption modifies Eq. (2) in that Eq. (2) contains the 
restriction that for the subject to be correct on the first 
response, he must guess correctly, therefore not allowing the 
possibility of being in solution (the learned state) on the 
first trial. Allowing for the possibility that the subject is 
im SOlution on the first trial [Atkinson, and others, 1965, 
footnote p. 55], appears to be a more realistic approach and 


was used in this work. 


1 Gaay, Bb. B.; Carrier, J. K., Jr.; Bradfield, R. a. sand 
Rigg, L., The Discrete Effects of Response Contingent Reinforce- 


ment During Skill Acquisition, submitted to Journal of A>plied 


Penavior Analysis tor publication. 
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oe)  hewagetaceer (guess) in presolution is a function of 
the aoa eleeene sequence (step) and the subject. 

4. The c factor (probability of going into solution) is a 
function of the step and the subject. 

5. gandc are constant over any step for a given subject. 


6. The set of outcomes form a homogeneous Markov chain. 


ier imal re P, (Correct | Row State) 
Le 1 0 L 
: (3) 
U, le ve=c) g 
Me Ours Slee 
Geo, Co) 
C. MODEL 


The equations developed in this work are a combination 
of what 1s presented in Refs. 1 - 3 and some original work. 
Since there is so much overlap it is not possible to give 
credit to any one source, instead the equations will be given 
with explanations and generally acknowledging the credit to 
all. The first development is the probability of errorless 
response given that the subject is known to be in the un- 
learned state (U). This state is assumed on the first trial 
and known EO Gast 1f an error voceurs. If @ subject is put 
on a program step and no error occurs before reaching the 
advancement criterion, then there is no way of determining 
whether he was in solution (learned state, L) initially, or 


was in presolution (unlearned state, U) initially and performed 


as follows: 


akZ 





Gate l—-c)e + pe ere) ate ts ers 


P(Errorless] 


~ Sam 


2 
4 
a os P({Errorless] noniGl=cy: (4) 


For future reference, this probability will be called p (rho), 
the probability of errorless response given that the subject 
is in presolution. (Being in presolution is demonstrated any- 
time a subject makes an error.) The above development says 
that the subject either goes into solution on the first trial; 
stays in presolution, guesses correctly and then goes into 
solution; stays in presolution twice, guesses correctly twice 
and then goes into solution; etc. The development indicates 
that the subject eventually goes into solution if errorless 
response is achieved after an error. The reader familiar with 
the Markov theory will note that -the term relating to remain- 
ing in presolution and having errorless response was omitted 
in developing Eq. (4). The omission was committed since the 
term g?(1-c)” goes to zero in the limit as n approaches infinity. 
The next development will be the expected number of errors 
given g and c. The development relies on the parameter o. The 
probability that the total number of errors is exactly k 
equals: 


k+1 


P[E = k] = é Noose) aaa) (eel se 


( 


lit 8 
oO 


1 
If this abbreviated mathematical statement were written out, it 
would represent every feasible combination of events in which 


exactly k errors can occur. 


L3 





k+1 


* PIE =k) = (1-g)*(1-c)¥e J (Sf*)g*(-c)? 


1=0 


Referring to a standard math table e.g. CRC [13th Edition, p. 


463]. This equation then reduces to 
k l k+1 
Pipe) “= SPC -gjti-c)) co ieenel=eir- 
fi-g) (1-c))* , ec 
I-g (1-c) ier (love 
k 


ne Cc c 
i T=g(i-ey! = 'W-gti-ey! 


Referring to Eq. (4) and substituting 


o.P[E =k] = (1-9)*p (5) 


In words Eq. (5) says that there were k possible errorless 
response Beings broken by the occurrence of an error and 
then there were no more errors. Since the probability of an 
errorless response string is op, given that the subject is in 
presolution, it follows that not errorless response is equal 
to (l-p). This takes into account all possible numbers of 
correct responses before the error response that breaks the 
String. The occurrence of an error demonstrates presolution 
and also allows another possible string of errorless response, 
which is independent of the length or number of previous 
Strings and is only dependent on the knowledge of being in 
presolution, 1.e., the occurrence of an error. In summary 
then Eq. (5) says that the probability of exactly k errors 
occurring is equal to not errorless response k times and then 


errorless response. 
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E(E]) = ) k-P[E=k] = J k-p(1-p)* 
, k=0 k=0 
= ation) alee? 
._ plli=-o) (1-p) 
= Fo 5 (6) 
0 
‘ptey = Ges) ee) 7 


The next development is the expected trial number of the 


last error. 


trial t equals: 


P[(T=0] 


P [T=t] 


t 


In words, Eq. (8) says 


The probability that the last error occurred on 


P 
(1-c) ©(1-g) p (8) 
i 2S, 28 9 


that there were t presolution trials 


indicated by an error on trial t and then errorless response. 


This probability statement 


number of correct and incorrect responses up to trial t. 


only required knowledge is 
then no more errors. 
(5) 


given by Eqs. and (8) 


are both decreasing functions. 


allows fLor=any sequence (onder) Sou 
The 


that an @Grror occurs on trial “teamed 


It should be noted that the probabilities 


This 


corresponds with what will be observed about Figs. 2 and l 


respectively. 
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E[(T] t>P(T=t] 


0 


Il 
li ~~ 8 


t 
co 


= .O.p+ ) t*P[T=t] 
t=1 


oO 


= J. t+(l-c) (1-g) 
t=) 


oO 


o(1-g) (l-c) } t+(1-c) 
t=] 


E(T] a 


Seo ic) 


2 
e: 


E[(E] 


Se ie = 
~ ¢ E(E] l-g(li-c 


) 


(deceSc) 


2 BNE, [l-g(i-c)Jc 


(10) 


With Eqs. (6) and (9) it only requires a little manipulation 


to get estimates of g and c: 


from Eq. (6) 


p = (E[E]+1)7+ 
From, EQ. 9(9) 
. SE 
ST o“g aca 
aaa = E[E] 
oC SOME TETD) 
ee. E (11) 
T(E+1) 


In Eq. (11) the expected number of errors E(E) is replaced 
by the observed number of errors and similarly the trial of the 


last error T replaces E(T). This might seem out of ord [xo. 
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some, in that a random variable is substituted for its expected 
value (mean or first moment). Some thought, however, will 
point out that the observed values of E and T are the mean 

when the sample size is one. Each subject/step combination 
represents a unique data point that cannot be repeated since 
the completion signifies the transition from the unlearned 
state (presolution) to the learned state (solution), and g and 
c factors are no longer applicable when the subject is in solu- 
tion. This procedure represents a special case of the applica- 


tion of method of moments in estimating the parameters of a 


austribution: 
1 E [E] 
~ ETT) 


Substituting E=E(E) and T=E(T) and utilizing Eq. (11): 


a 





.Q > 
il 


1 - E 

T(EFL) 
_ (T-E) (E+1) 
- T(E+1)-E 
Z 


e “A E 
re = 1 - T(EFI)-E (12) 


If E is much greater than 1 (E>>1) then 


as 1 
Cc = — 


Als 


Also 


4sQ > 


Ae on OMENS 
lb 
(T-1) + E 


pemecemieis greater than or equal to E(T > E) 


+3] GJ 


g>1- 
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These equations state that c is approximated by the reciprocal 
of the number of the trial of the last error. This is intu- 
momety appealing aS 1t States the larger the c factor 
(probability of going into solution), the fewer expected number 
of trials. The estimate of g is slightly larger than the 
apparent presolution accuracy. This comes from the concept 
which allows the possibility of sparetailysinevern correct responses 
after the last error. If the number of these responses were 
ia@ewm, the estimated presolution accuracy rate (g) would 
increase above what can be estimated using only the trials to 
the last error. The required correction factor is taken into 


account in Boe (2)... 


PREFACE — SECTION I1iI 


During the final proofreading of this thesis it was discovered that 
the computation of g (Appendix B and Figs. 4, 7, and 8) had been done 


2 2 


; A | E 
instead of g = 1 T(Et1) -E 


E 


using: Gq: = l- T(Et1) +E ’ 


(Re. sd 2) 


The second equation is the valid equation as it appears in the text. 
Specific reference to g, as it applies to the data analyzed, must therefore 
pe considered in light of the first equation, e.g., pages 20 and 21. 


The effect of using the first equation is to cause an across-the-board 
increase in g. The general conclusions are still valid but the spec‘ fic 
conclusions as regards the data analyzed, must be viewed in light of this 
new.information. Examples of the effect of this change are: (1) a ‘eneral 
shift of Figs. 4, 7, and 8 to the left, (2) a general decrease in the mean 
values of g listed in Tables I and II, (3) a general decrease in the esti- 
Mated values listed in Table III. This is based on observing that the 
Slope of the associated equation (page 25, no number) is positive with 
respect to g. This would move the estimated value toward the observed 
(12% estimated and 8% observed), and (4) a general increase in the esti- 
mated values listed in Tables IV and V. This is based on observing that 
the slope of Eqs. (15) and (16) are negative with respect to g. This would 
move the estimated values toward the observed. 
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Le See etCATIOn OF THE MODEL 


= DATA SOURCE 

With the estimates of g and c being obtainable from only 
a knowledge of the total number of errors and the trial num- 
ber of the last error, it was now a matter of obtaining some 
data to evaluate the model. The Institute has 41 programs in 
operation. These programs were being used in 20 different 
locations in the country. Due to the nature of the overall 
program, individual students receive only those programs which 
they require. It was therefore difficult to determine which 
program to sample. After some consideration the "What is" 
Paogram was decided On. Calls were sent out to four Sites for 
complete data on students having completed this program. There 
were 19 subjects from the 4 locations, the least from any loca- 
tion was 2 and the most was 7. There were 11 boys and 8 girls. 

The “What is" program consists of 35 steps grouped into 9 
series. With 19 subjects there was a possibility of obtaining 
(35x19) = 765 sets of data points, each representing a unique 
student/step combination. iro temo the remainder of this 
work "data point" will refer to a set of data from a unique 
Student/step combination. Only 208 data points were available 
for the following reasons: 

1. Due to the placement procedure the student may 

Start at a point other than the beginning of the program. 


(There were 208 missing points due to this.) 


a9 





2. If a student is put through a step and makes no 
errors, there is no way to tell whether he was in solution 
before he started or had an errorless response string. 
Therefore, these points yielded no usable information and had 


to be ignored. (There were 349 of these points.) 


B. EXAMINATION OF DATA 

The raw data formed a 35 by 19 matrix with 208 data points. 
Each data point consisted of the trial number of the last error 
(T) and the total number of errors (E). The data was then 
converted into a Similar matrix containing c and 5 at each 
data point using Eqs. (11) and (12). The first step was to 
examine the various frequency distributions to see what informa- 
tion they would yield. Frequency distributions are presented for 
the Ranier of trials (T), number of errors (E), c¢ and g. These 
distributions consisted of a total of 208 data points each and 
are shown in Figs. 1, 2, 3, and 4 respectively. Figure 1 shows 
the frequency distribution of the trial number of the last 
error, the interval widths are two trials. Figure 2 shows the 
frequency distribution of the number of errors, the interval 
widths are one error. Figures 3 and 4 show the frequency 
distribution of ¢ and g respectively, the interval widths are 
04, Figures 1, 2, and 3 exhibit a consistent decreasing 
nature, which indicates the lower values of T, E, and Cc are 
predominant in the sample and should generally be expected if 
the sample is representative of the population. Figure 4 has 
a much different shape, somewhat similar to a truncated normal 


type distribution. Figures 3 and 4 each have one point hat 
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breaks the otherwise consistent pattern. These points are 

in the interval .48 to .52 for C, ana 64 to. .68 for g. There 
appears to be a relatively simple explanation for these points: 
Data points having 1 error and 1 trial in presolution, that is 
an error on the first trial only, yields a c= ~50 9 andsa g = 
.667. If a student was in solution, i.e., knew how to 

Seaver correctly, but misunderstood the protocol the chances 
are good that he would have one error only. This protocol 
concept seems a valid explanation for the cut of place peaks 
in Figs. 3 and 4. 

Examination of the individual data (Appendixes A and B) 
showed no consistent trends. All of the factors T, E, C and g 
vary both over subjects and over steps. Accordingly the next 
approach was to calculate the means and standard deviations of 
¢ and G by subject and by step. The results are shown in 
Tables I and II, respectively. The frequency distributions of 
these values are plotted in Figs. 5 through 8, Figures 5 and 
6 show c by subject and step, respectively. It is apparent 
that these frequency distributions overlap and that they fall 
under the overall frequency distribution of c shown in FE PGs 
Similarly Figs. 7 and 8 overlap and fall under the overall 
frequency distribution of g shown in Fig. 4. 

It had been hoped that some definite consistency in one of 
the parameters would show up over subjects or steps. It appears, 
however, that the parameters g and c are random variables from 
the overall distributions shown in Figs. 3 and 4. Assuming 


that the sample is representative, it then follows that n this 
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TABLE I 


MEAN AND STANDARD DEVIATION OF g AND c BY SUBJECT 
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Program the values of g and c, and therefore L and E, are 
random variables from the general populations indicated by 


Figs. 1 through 4. 


C. CRITERIA DETERMINATION 

Based on this conclusion, general criteria for all students 
on this program can be determined. Two criteria have to be 
established: (1) At what point in the program can the student 
be advanced with a high probability of being in solution? 
(2) At what point in the program should a student be sent 
to a subroutine which has lower difficulty levels? 

1. Advancement Criterion 

The first criterion is based on the student's g level 

for that step and can be analyzed from the individual data 
given in Appendix B and Fig. 4. “The individual data (Appendix 
B) showed that the largest g for this population is .974. Using 
this aS an indication of the highest g that might be expected, 
the next step is to determine its approx. probability of occur- 
rence [P(g=M)]. Figure 4 shows that there were 8 times when g 
waS recorded between .96 and 1.0. This represents approximately 


-04 or 4 percent. 
Peeters ote 204 


Table III gives an indication of what the probability 
would we of not being in solution. The values listed are for 
several different consecutive numbers of correct responses 
(X) as the criterion. M is the midpoint value of g used in 


Hemet .e., (.98, .94, .92..... ) and is used to repres nt 





TABLE III 


ESTIMATED PROBABILITY OF NOT BEING IN SOLUTION 
[P(S)] ONCE THE CRITERIONOF X CONSECUTIVE 
CORRECT RESPONSE HAS BEEN ACHIEVED 





Gra terion Estimate 


X P(S) 
8 26 
10 | 21 
12 17 
14 14 
16 ay: 
18 sie 


the interval. The following basic, conditional probability 
statement was used for these calculations. 


P[Not Solution]=P[S]=)P[Not Solution|g=M,Criterion=X] P[g=M] 
M 


+ P(S]=) (M)*+P[g=M] 
M 


It 1S obvious that the criterion must be established 
based on the desired certainty. For purposes of validation, 
let the criterion for completion be established at 16 con- 
secutive correct responses. The actual raw data criterion 
levels of 10 and 20 consecutive correct responses were used 
in a rather arbitrary fashion. Of the possible 765 data 
points, there were 765 - 208 = 557 in which the subject was 
tested. Of these 557 data points, there were 306 which «sed 


a criterion level of 20. Of these 306 data points, thei: 


INS 





were only 24 in which a criterion level of 16 would not have 
been satisfactory. That is, there were 24 data points where 
16, 17, 18, or 19 consecutive correct responses were recorded 


followed by at least one more errors. This value 


24 2. 
ais. a .08 = 83% 


means that 8 percent of the time the criterion of 16 was in- 
adequate. The prediction was 12 percent, and therefore the 
method of setting this first eceron level seems appropriately 
conservative. | 
2. Branching Criterion 

To establish the second criterion we turn to Figs. 1 and 
2. Figure 2 tells us that 90% of the data points had less than 
25 errors. Figure 1 tells us that 87% of the data points were 
completed in less than 90 trials. What is more significant 
about Fig. 1 is that the data points which had more than 90 
trials required a total of 5065 trials, whereas those requiring 
less than 90 trials required only a total of 3988 trials. This 
means that 13% of the data points required approximately 56% of 
the effort. Similarly, the upper 10% of the data points showed 
1586 errors out of a total of 2575 error. This means that 
approximately 61% of all the errors occurred in 10% of the data 
point. These two observations indicate that there should be 
some upper value of E and/or T which would signify the need 
for branching. That is, some point that, if reached, would 
indicate that the subject is having an exceptional amount of 


difficulty and should be branched. To develop this furt! 2r 
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mequires returning to Eqs. (5) and (8). 


that these equations are 


Rim 
k->00 


P [E<k] 


Qim 
{700 


P[T<t] 


Based on 


P [E>k ] 


ee P[E>k] 


The following shows 


true probability mass functions. 


(eg) (1-c) )** 


) p(1-p)? 
i=0 


Peo (Sapni-eiir 
owe 


ptp(1-g) (1-c) J (1-c)*7+ 


i=l 
1 

ptp(1-g) (1-c) = 
0 Cope A (1=c) ) 


c+l-c-g(l-c) 
a 


) 


l-g(l-c) 
aa 


il ul 
¢) au = 1 


this development it follows that 


K : 
1 - } p(1-)* 
1=0 


Weallekweojt | 


Toe (eases 


JL 


Tec) ee 
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and 


c . 
ieee pep (clean alee) 


P(T>t) = 
1=1 
a i-l 
= (l-o)- p (1- g)(1l-c) } (1-c) 
i=l 
ec) 
= (1-p) -p (1-g) (1-c) —————— 
as (l-g) (l-c) a (I1-g) (l-c) (Wet ae Sy 
I-g (I-c) l=g (1-c) 
“.P(T>t] = etsy ti-c1* (14) 


With Eqs. (13) and (14) in hand, it simply becomes a matter 
of calculations. The calculations are similar to those done 
forme the first Criterion. Tojwget the unconditional probability 
Eeet £ Or T is greater than some number requires the following 


basic type equations, involving two independent random variables 


igmand Cc). 
P(E>k]) = )) plE>k|g=x,c=y] P[g=x] P[c=y] 
’ gC 
k+1 
= )) anes P [g=x] P[c=y] (15) 
and 
P[T>t} = )} P(L>t|g=x,c=y] P[g=x] P[c=y] 


gc 


a (1-x) (l-y) 
» ig 


= [1-y]“ P[g=x] Plo=y] (16) 


Tables IV and V represent the results for E and T respectively. 


PMso fisted in Tables IV and V are the values obtainec om the 
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lower abscissa's of Figs. 2 and 1, respectively. The lower 
abscissa's represent the cumulative percentages, that is, the 
percentage of data points with values less than or equal to 

the value indicated on the First abscissa. For the sake of 
comparison, one minus the values in Figs. 2 and 1 are used 

in Tables IV and V. This value then represents the percentage 
of data points in which E and T were greater than the associated 


criterion number. 


TABLE IV 


ESTIMATED PROBABILITY THAT THE NUMBER OF ERRORS 
WILL EXCEED A GIVEN CRITERION k [P(E>k) ] 


Criterion Estimate Observed 
k P (E>k) Figure 2 
5 RAS) . 36 
10 ley s/o 
Ws sAkZ aro 
20 .08 14 
25 205 SO 


(Also associated with this table are the observed 
percentages of data points which had more than k 


SErOrs | Erom F1lgy 2) ) . 


It should be noted that the observed values in Tables 
IV’ and V are consistently higher than the expected values. 
Some thought indicated that the asceneeion of independence, 
required for the development of Eqs. (15) and (16) may nt 


have held up. A Pearson "r" test waS run with g=X and: Y. 


Zo 





TABLE V 


poles D PROBABILITY THAT THE NUMBER OF TRIALS 
WILL EXCEED A GIVEN CRITERION t [P(T>t)] 


Criltervon Estimate Observed 
Ie P(T>t) Figure l 
60 om ie .20 
70 ~ 10 Alte: 
80 5 te ~ 16 
90 OFT alle: 
100 06 palez 


(Also associated with this table are the observed 

percentages of data points which had more than t 

Emails [from Fig. 1]). 
The result was r=-.2774 with n=208. This value iS significant 
at io than the P=.01 level. With a knowledge of this fact an 
examination of Eqs. (15) and (16) indicates why the estimated 
values were lower than the observed values. It is felt however 
that the correlation between g and c is an unavoidable Renee 
of the estimators. In explanation of this last statement, 
observe that Egs. (11) and (12) have an upper and lower bound 
based on the value of T. These bounds are most restrictive at 
the lower values of T where ¢c is larger than g has its most 
restrictive upper bound. Since 36% of the population had a T 
value of ten or less (Fig. 1), it 1S relatively easy to accept 
a higher ¢ and lower g (negative r) bias in the estimators. 
The branching criterion must therefore be established wit an 


awareness of this artifact. 
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One case of branching was observed in the raw data. 
At that time the branching criterion required three consecu- 
tive sessions with a percent of accuracy less than 80%. It 
took 184 trials before branching was performed. Had a 
criterion of 25 errors been used, the subject would have 
branched after the 57th trial, which would obviously have 
been an advantageous move. Since there is no apparent way to 
Measure the benefit of branching over just forcing the subject 
through the step, a comparison similar to that done for the 
first criterion cannot be made. It is felt however that the 
confidence on using the number of errors or the number of 
trials as branching criterion will lie somewhere between the 


estimated and the observed values (Tables IV and IV). 


oul 





IV. CONCLUSION 


In general, any program that has the form of those used 
in this study could be analyzed in a similar manner. Once 
the program is ready for testing, a representative sample 
should be run through with an arbitrarily high criterion, Say 
20 consecutive correct responses with no branching. (Branching 
changes the basic probabilities (g onl @) and confounds ‘the 
data.) Analysis of the resulting data would then indicate 
the criteria to be used when the program iS given general use. 


For this particular program the following criteria seem adequate: 


Pass Criterion = 16 consecutive correct responses 
(88-92% confidence) 


Branch Criterion = 25 total errors (90-95% confidence) 


or Branch Criterion = 100 total trials (88-94% confidence) 


It 1S recommended that anyone using this procedure attempt 
to increase the number of data points at least twofold. This 
increased number of data points would smooth out the frequency 
distribution and possibly point more exactly to how c and g are 
distributed. Having a smoother frequency distribution wiil 
also fill in blanks in the joint probability matrix used in 
calculating P(E>k) and P(T>t). With more cells filled in, it 
may be possible to use the actual joint density P(g=X, c=Y) 
rather than the product of the marginal densities P(g=X) -P(c=Y). 
This would bring the estimated and observed values in Tables 
IV and V closer together because it would remove the requ re- 


ment for an assumption of independence which appears to 
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weeotrateddue to the model. Knowledge of this procedure 
before the data is taken will also allow for better control 


of the data gathering. 
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APPENDIX A 
Individual Data Matrix (T and E) 
Rows Represent Subjects, Columns Represent Steps 
Tep °aNtry Bemals T, Bottom Entry Equals E 


Subject A A A A A A A B B B B B 


12 71 47 0 0 0 0 0 0 0 0 0 


: 4 42 18 0 0 0 0 0 0 0 
> tu 0 0 0 0 0 0 0 0 0 0 0 
| 0 0 0 0 _0 0 0 0 0 0 0 

3 = = = = - = = = 2 = = 
4 a = = =< a = = = = = = = 
5 0 0 9 0 0 0) 0 0 0 0 0 0 
0 0 3 0 0 0 0 0 0 0 0 0 

6 0 2 1 0 3 0 0 0 0 0 
0 2 1 e) uf 0 0 0 0 0 0 

7 = = a = am x = # = = = = 
8 af = = = = = ss = = = = 
9 4 0 0 0 2 0 0 0 0 0 0 0 
0 0 0 2g 0 0 0 0 0 0 0 

10 0 a ue 8 8 at 710° FLOSS we 219 4¢ oo 62 





34 








APPENDIX A 


Individual Data Matrix (T and E) 
Rows Represent Subjects, Columns Represent Steps 
ZOp Entry Equaeis FT, BottomeEntry Equals E 


mime Cc C¢ C, C¢, GC BD, DP, Dy Dy, Dz, CB, 





i 0 0 O 0 O O 0 
0 0 0 0 0 0 e) O 0 
9 0 0 39 14 10 24 
0 0 12 2 2 5 
: eee 7, Oe 8. AO Vo 
14 O O 3 0 0 
A . 3 22 O O 9 0 0 15 25 0 
3 1 8 0 0 0 0 4 i3 0 
5 0 0 0 0 0 3 39 2 4 0 
0 0 O O O 1 5 i - 0 
6 2S, 2 O r2 0 O 0 0 22 
3 2 0 7 0 0 0 O 3 
7 a = = z 2 = = 7” = a sf 
8 a = = = = = = = = = = 
41 0 0 O 0 14 3 41 0 6 0 
: 7 0 O O il al 18 0 5 0 
10 6 0 43 30 0 26 85 15.221 0 16 


Sys. 





-APPENDIX A 


Individual Data Matrix (T and E) 
Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 


Subject EA Ey Eo Fy F. FP, G) G, G, Hy Ty 


1 ws 
0 50 4 
: 0 O 0 0 0 0 0 0 
: e) a O 0 69 0 8 
2 4 0 20 1 
P 35 aw 2a 0 y Cn CaOun 23 O° toe 
5 5 O i] 6 6 1 7 
z 0 0 Oo 3 3 0 3 2 0 
0 0 ) 3 1 1 i i 
- 10 0 0 D. 4 0 10 35 0 12 
3 0 ) 1 1 0 4 6 0 3) 7 
: : 7 _ : 0 ) wae 6 : 7 
0 O 1 3 2 
8 a = = = = = i = 
0 0 O 
0 ) 0 0 55 47 16 0 
fe) 0 0 0 0 O 5 5 2 
a 2 57 0 5 43 37 Ol selel ile als : 
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APPENDIX A 


Individual Data Matrix (T and E) 
Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 


Puesect 75 AW AG UA, COU CUA OAS By ee eS) 


a 0 1 1 0 0 O 0 0 15 
iT 1 6) @ © 0 oO il 
i 0 3 0 0 91 48 178 O 3 3 0 
0 2 0 oO 34 9 67 0 1 1 0 
be 32 138 277 12 18 0 21 59 gis 
8 85 123. 2 2 03 Ts 1 5 
14 
i 
15 x ee z = Sa ee = = 
- 14 oO 1 96 9 Z 0 38 106 O 
0 3 Gi fi 9 2 il 2 19 
= 1 i 2 
i 1 1 0 Oo oO 
re 44 il 0 0 Cm so 6 0 0 0 0 
ion) al 0 0 Ome at0r 0 0 O 0 
neh Cao 10 1 0 0 EG 
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APPENDIX A 


Individual Data Matrix (T and BE) 


Rows Represent Subjects, Columns Represent Steps 


Top Entry Equals T, Bottom Entry Equals E 


Subject C, 
11 10 
iL 
12 eg 
10 
13 170 
93 
14 2 
0 
B ga) - 
16 : 
0 
7 e 
0 
18 : 
0 

19 


36 
nae 


269 
99 


28 


16 


10 


38 


3 


233 
78 


2) 


sil 


D 


2 


a3 
24 


25 


23 


pe 


262 
a7 


235 
40 


55 
17 


54 
14 


78 
16 


o2 
24 





‘APPENDIX A 


Individual Data Matrix (T and E) 
Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals T, Bottom Entry Equals E 


Subject E E E Les 2 E G G G H I 





ll 0 0 0 17 e) PAA 0 8 0 0 
0 0 1 e) 2 0 1 e) 0 
12 lew LOL 12 QO 142 zi 2 13 4 0 0 
4 15 3 0 32 it 3 3 1 0 0 
13 12 80 49 47 178 104 23 64 334 0 88 
2 18 al a7 44 Lo 2 24 112 0 aly 
14 0 0 e) 9 0 1 86 6 

0 0 0 0 1 0 ue 14 0 0 

15 70 6 1 oy 0 ) 13 184 0 82 
22 3 3 4 e) 3 4 86 0 32 0 
16 22 0 0 50 0 0 0 
0 0 0 2 0 0 2s e) 0 0 

17 0 0 0 0 0 0 36 41 0 6 
0 0 0 0 0 6 3 0 i 0 
18 0 0 0 0 0 0 75 36 0 0 0 
0 0 0 0 0 0 15 3 0 0 
19 0 0 0 0 0 0 
0 e) 0 0 0 0 0 0 


Sy, 








Individual Data Matrix (g and c) 


APPENDIX B 


Rows Represent Subjects, Columns Represent Steps 


Top Entry Equals g, Bottom Entry Equals c 


(S = Tested with No Errors, S' = Not Tested) 
Subject A, A, A, A, A, Ag A, B, By, Bz, By 8B, 
: 750 .570 .644 S S S : S S S = 5 
.067 .014 .020 
2 ae & S S S S S S S S S S 
~L25 
3 wesc es Sta as’ “Stee st st stee st §* -¢E 
4 aS ec wot. bs’) S's! “St gyre *s’ st “4s! 
.770 
5 S 5 og3 § S S S S S S S S 
6 Bote 2b omaer | gi 263 «g S S S S S S 
< 333 S00 038 
7 SS ee) s* Ste S's) Sto Sst Ss" st) CC S* Sst 
8 clases s' s® s' os" st ost s' s' s! sg! 
9 wee GS S Sa aSemeis S S S S S 
. 200 333 
le g +067 .667 .941 .941 .956 .843 .871 .818 .907 .760 .745 
.500 .500 .063 .063 .045 .012 .009 .004 .019 .010 .015 
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APPENDIX B 


Individual Data Matrix (g and c) 
Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals g, Bottom Entry Equals C 
(S = Tested with No Errors, S' = Not Tested) 


Sebject ¢| Cc, ¢, €, C Dy Ds Dy Db, De OB, OC, 





woe alo LO 
wO2Z5 ,023- .008 


: ec oeecol crs. GR. £832. 
.024 .048 .067 .035 
: Tee, Comes) 4 GT 
025 094 500 
h ee so7iees) TCM SI 
.150 .167 .040 .093 -0S8m037 


.857 .845 .800 .889 
eles U2. ZOO nL eo 


~943 .500 .924 908 


6 S sec Ss: Sj) Cans 
.019 .333 .073 .034 

7 se sts" 3s' st “st (s* Gime Gai s'aes* -s* 

8 ees’ ce «GH s* s'@ic' Som 6 js" ss" 5! 

: fee ee 2 26557 595) a co0 me ye 
021 .036 .125 .023 139 
e e e e 848 e e @ e 

if g00 . .905 .896 . 809 915 .802 , 866.800 
111 019 .027 .033 .011 .044 .004 .047 .250 
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-APPENDIX B 


; Individual Data Matrix (g and Gc) 
Rows Represent Subjects, Columns Represent Steps 


Top Entry Equals g, Bottom Entry Equals C 


(S = Tested with No Errors, S' = Not Tested) 

Subject ES E, E. F) ES F. G) G. G, Hy 
1 S S S S S S coe S S ae 
.009 OLS 

2 S S S S S S S S S S 

3 S S soo2e 4886 S 2O73 S 728 .941 S 

074 .061 069 014 .063 

4 .884 S -850 S aos ae S 526 .784 .800 
.024 2031) -O71 .063 nOSo 2037" 1250 
5 S S S S (ote «L857 -857 S 2397 2600 
-044 .167 .167 2167 9,250 
6 roll S S -800 .889 S ;704 .857 S s700 
075 720) Welz 080 .024 .042 

5 & a 7 SI 667 .834 .800 ., 

900 .063 .111 

8 s' Ss! Ss! S' Ss! Ss! S S S Ss 
9 S S S 5 S S S 2925 2913 2.920 
s0lS” .01e .0aZ 
10 500 .792 S 7909 “2656  .890 S -/40 2/7/17. 2866 
Fogo »O0L6 lOO” 20212) .023 .003 .008 .047 
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.063 
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APPENDIX B 


Individual Data Matrix (g and C) 
Rows Represent Subjects, Columns Represent Steps 
Top Entry Equals ce Bottom Entry Equals C 
(S = Tested with No Errors, S' = Not Tested) 


Subject A A A A A A A B B B B B 





mao ee TG ls lhlUcSlUCGlCUCCSlUClC me OOP 
5200 2500 2033 
e636 =, «4 «834 631. «857 B57 
, 222 .O11 .019 .006 ea elle? 
m .784 ,396 .561 .895 .929 , .897 .765 .941 .699 .909 
.028 .007 .004 .056 .037 .036 .016 .063 .064 .100 
wee ss 6 Ss Ss & “S 5S Ss 5 
500 
i  s' SSE es) ISueCE) «OCH 6m) Sst agi 
1% 06 gt 847 g~— 067-814 .862 800 , 966.831. 
054 .500 .010 .074 .167 018.009 
col) eae COO eg 
.500 .500 250 
‘en Se & © 'S - 5 ~8 | GOS se: 
.021 .500 
ior S SMS wom sc es: ts oe 
500 500 
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APPENDIX B 


Individual Data Matrix (g and ¢c) 


Rows Represent Subjects, Columns Represent Steps 


Top Entry Equals g, BeoteonmeEntry Equals C 


(S = Tested with No Errors, S‘' = Not Tested) 
Subject C, C, CC, C, Cy, D, D, D, Dy, De EB, Ey, 
ae}. Se S S S S S S S S i S S 
.050 026 
12 sJ05-.754 .637 .923 <516 S S S .648 S gtOo1ee 13 
-045 .025 .004 .083 .005 .003 .004 .017 
13 -462 S .889 .970 S Oster 5 73) arom S ~811 .709 .762 
.006 029 .03t .004 .018 .024 f00s .007 .017 
14 S S S S S S S S S S S S 
4s ai ep gh eo ego ONS ey 280 BECCA .815 .809 
032, .,08e@ .139-.029 .O21 .O12 
16 S Py 20) S S S .809 S aa S .889 e 565 
e107 .068 e025 O18 
17 S S S 947 .875 e7O0 S S S S S 
.056 .067 053 
18 S S S S S S SEO NSE S S S S 
2033 <2 
19 °S S S S S = S S S S S S 
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- APPENDIX B 


Individual Data Matrix (g and c) 


Rows Represent Subjects, Columns Represent Steps 


Top 
(S = Tested with No Errors, 
Subject E, E, E, F, F, Fz; & 
7 é P E 27 ae 7228 
029 Oe 
12 810 .862 .824 1783 946 .770 
050 .009 .063 .007 .O7 .083 
ie .895 .789 .798 .665 .760 .828 .944 
056 .012 .019 .020 .005 .009 .029 
a a : « -047 same. CO7 
.056 .500 
1 703 .667 .770 .915 . .770 .768 
0248 .125 083 .022 .083 .062 
16 S S S s + tara S 
~030 
17 S Ss Ss S S S 28k 
024 
18 S S é S S 5 
role 
19 S S S S S S S 
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.836 
2028 


.645 
705 


850 
-OL1 


.540 
e 005 


ood 
SOL? 


i 
O28) 


.949 
AS Pah 


Entry Equals g, Bottom Entry Equals C 
S' = 


Not Tested) 


See 
2125 


689 
.003 


923 
083 


626 
sOrZ 


e925 
20383 


® 819 
O21 
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